Skip to content

Improve CI Docker cache reuse#1296

Merged
lovasoa merged 16 commits into
mainfrom
improve-ci-cache-and-dedup
Jun 3, 2026
Merged

Improve CI Docker cache reuse#1296
lovasoa merged 16 commits into
mainfrom
improve-ci-cache-and-dedup

Conversation

@lovasoa
Copy link
Copy Markdown
Collaborator

@lovasoa lovasoa commented Jun 2, 2026

Summary

  • Reuse the Linux test binaries and Playwright binary produced by compile_and_lint instead of recompiling sqlpage in the database and browser-test jobs.
  • Keep Windows incremental/local sqlpage artifacts in a PR/ref-scoped cache so source edits can reuse incremental compilation state without allowing PRs to write a trusted main cache.
  • Make Docker cache scopes explicit: PR caches write to PR-scoped BuildKit GHA scopes, while trusted main remains a restore-only Docker Hub fallback for PRs.
  • Move .cargo/config.toml into the Docker dependency warmup layer so the warmup and final build use the same LIBSQLITE3_FLAGS.
  • Use BuildKit's Git context for Docker builds so dependency layers are keyed by committed source rather than by a per-job local runner context stream.

Current findings

  • The Rust job deduplication works. In CI run 26915251134, the DB matrix and Playwright jobs used uploaded binaries and did not run Cargo builds.
  • The Docker final-build warmup gap is fixed. In 26892512893, amd64-minimal build-project.sh compiled only sqlpage; the previous repeat compiles of libsqlite3-sys, sqlx-core-oldapi, sqlx-macros-oldapi, and sqlx-oldapi disappeared after copying .cargo/ before build-dependencies.sh.
  • The remaining Docker dependency-cache miss came from using context: .. Same-commit reruns had identical Docker frontend args and checkout SHA, but different BuildKit local-context vertex digests, which invalidated every downstream COPY and therefore build-dependencies.sh.
  • Switching Docker builds to context: "{{defaultContext}}" fixes that instability. The first Git-context build is cold once per platform because the old path-context cache keys are not reusable. After that warmup, same-commit reruns hit: amd64-minimal rerun job 79405292019 completed in 23 seconds, and armv7-minimal rerun job 79406759422 completed in 24 seconds, with build-dependencies.sh and build-project.sh both CACHED.

Validation

  • git diff --check
  • actionlint -ignore 'SC2046' .github/workflows/ci.yml
  • docker buildx build --check --target builder .
  • CI run 26915251134: full workflow passed.
  • CI run 26915251134, rerun jobs 79405292019 and 79406759422: same-commit amd64-minimal and armv7-minimal reruns proved the Docker GHA cache hits the dependency layer after the first Git-context warmup for that platform.

Reverted experiments

  • BuildKit Cargo cache mounts were tested and reverted earlier. On same-SHA rerun they still downloaded hundreds of crates and compiled hundreds of units, worse than plain layer caching.
  • A dummy lib-target warmup was tested and removed. It did not stop SQLx recompilation; copying .cargo/config.toml before dependency warmup did.

@lovasoa lovasoa force-pushed the improve-ci-cache-and-dedup branch from 5ecd502 to 0adf3d4 Compare June 2, 2026 15:31
@lovasoa lovasoa force-pushed the improve-ci-cache-and-dedup branch from 0adf3d4 to 3c615a0 Compare June 2, 2026 15:35
@lovasoa lovasoa marked this pull request as ready for review June 3, 2026 12:20
@lovasoa lovasoa merged commit 8b9d6a0 into main Jun 3, 2026
153 checks passed
@lovasoa lovasoa deleted the improve-ci-cache-and-dedup branch June 3, 2026 22:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant